Skip to content

feat(catalog): Catalog with an optional multi-index hashing (MIH) index - #65

Merged
mrecachinas merged 1 commit into
perf/raw-fastcallfrom
feat/catalog-mih
Sep 23, 2026
Merged

mrecachinas merged 1 commit into
perf/raw-fastcallfrom
feat/catalog-mih

Conversation

@mrecachinas

@mrecachinas mrecachinas commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Adds hexhamming.Catalog(records, width, index=False). It copies and validates a fixed-width catalog once, then serves first/best/all queries (single and many). Results, sentinels and tie-breaking match the free functions.

index=True builds a multi-index hashing (MIH) index:

  • m flat CSR bucket tables, each s ≈ log2(N)+2 bits (capped at 22), with u32 record ids.
  • A query probes every key within t = max_dist // m bits of each query substring. By pigeonhole this finds every record within max_dist.
  • Each candidate is verified with the byte kernel.
  • Candidates are not deduplicated or sorted: ties go to the lower record index directly, and only all() sorts its matches. This avoids a per-query N-bit seen-set.

Planner. For each query, a cost model chooses between the index and a linear scan. It compares probes × per-probe cost (which rises with index size) against records × scan cost.

  • It was fitted to a sweep of every probe radius 0..=4 on 8/16/32-byte catalogs of 10k–1M records.
  • It chose the faster path in all eight near-crossover cases measured; mispredictions elsewhere can only favour the linear scan.
  • Python calls borrow exact bytes queries and detach the GIL only when the estimated work is ≥ 1 µs, so small catalogs cost the same per call as the free functions.

Apple M4 Max, 1M records, best_within, one query:

Width r Free function Indexed Speedup
8 B 2 114 µs 82 ns 1392×
8 B 8 114 µs 1.29 µs 88×
8 B 12 108 µs 56.4 µs 1.9×
32 B 8 178 µs 144 ns 1237×
32 B 32 177 µs 5.77 µs 31×

Index build takes 24 ms (8 B) / 91 ms (32 B). Memory is Σ 4·(2^s_j + 1 + N) bytes: 45 MB / 182 MB. A 64×16 B catalog without an index costs 59.8 ns per call vs 58.9 ns for the free function.

Tests

  • Differential tests against the free functions across widths, radii and many-query variants.
  • A test that drives the indexed paths directly at every probe radius 0..=4, with planted ties and repeated candidates, so the index path is covered even where the planner would scan.
  • Python tests for the class API, validation and the README examples.

@mrecachinas
mrecachinas added this pull request to stack #60 September 23, 2026 02:22
@github-actions

Copy link
Copy Markdown

📊 Benchmark Comparison Results

✅ All benchmarks within noise

104 benchmarks compared, no significant changes detected.

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

hexhamming.Catalog(records, width, index=False) copies and validates a
fixed-width catalog once and serves first/best/all queries, single and
many, with the free functions' results, sentinels and tie-breaking.

With index=True it builds an MIH index: m flat CSR bucket tables of
s ~ log2(N)+2 bits (capped at 22), with u32 record ids. A query probes
every key within t = max_dist // m bits of each query substring. By
pigeonhole that finds every record within max_dist; each candidate is
then verified with the byte kernel. Candidates are not deduplicated or
sorted. Ties go to the lower record index directly, and only all()
sorts its matches. This avoids a per-query N-bit seen-set, which glibc
would also have to zero.

For each query, a cost model picks the index or a linear scan: probes
times per-probe cost (rising with index size) against records times
scan cost. It was fitted to a sweep of every probe radius 0..=4 on
8/16/32-byte catalogs of 10k-1M records. It chooses the faster path in
all eight measured near-crossover cases; mispredictions elsewhere can
only favour the linear scan. Python calls borrow exact bytes queries
and detach the GIL only when the estimated work is at least 1 us, so
small catalogs cost the same per call as the free functions.

Apple M4 Max, 1M records, best_within, one query:
  width  r   free fn   indexed   speedup   (build: 24 ms / 91 ms)
  8 B    0    13.9 us   85 ns     164x   (exact match early-exits)
  8 B    2   114 us     82 ns    1392x
  8 B    8   114 us    1.29 us     88x
  8 B   12   108 us    56.4 us    1.9x
  32 B   8   178 us    144 ns    1237x
  32 B  32   177 us    5.77 us     31x
  64x16 B catalog, no index: 59.8 ns vs free function 58.9 ns

Index memory is sum_j 4 * (2^s_j + 1 + N) bytes: 45 MB for 1M x 8 B
and 182 MB for 1M x 32 B.

Tests: differential tests against the free functions across widths,
radii and many-query variants. A new test drives the indexed paths
directly at every probe radius 0..=4, with planted ties and repeated
candidates, so the index path is covered even where the planner would
scan. Python tests cover the class API, validation and the docs
examples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
@mrecachinas
mrecachinas merged commit 2cc9cfb into main Sep 23, 2026
31 checks passed
@mrecachinas
mrecachinas deleted the feat/catalog-mih branch September 23, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant